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ABSTRACT 


In  this  paper  we  study  a  gradient  projection  method  for 
quadratic  programs  that  does  not  require  the  generated  points 
to  be  feasible  and  can  avoid  the  computation  of  a  feasible 
starting  point.  This  is  done  by  using  an  exact  penalty  func¬ 
tion  in  the  line-search.  It  is  shown  that  the  method  can  pro 
duce  from  any  starting  point  a  solution  in  a  finite  number  of 
iterations.^. 
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SIGNIFICANCE  AND  EXPLANATION 
Many  nonlinear  optimization  problems,  such  as  general 
nonlinear  programming  and  minimax  problems,  can  be  tackled  by 
solving  a  sequence  of  quadratic  programs.  Therefore,  it  is 
very  important  to  develop  efficient  computational  method  for 
quadratic  prog  rents.  For  such  problems,  we  propose  a  gradient- 
projection-type  method  that  does  not  require  that  the  generated 
points  be  feasible.  By  so  doing ,  we  can  avoid  the  expense  of 
computating  a  starting  feasible  point.  The  method  is  shown  to 
attain  a  solution  in  a  finite  nunber  of  steps  and  this  has  led 
to  satisfactory  computational  results. 


The  responsibility  for  the  wording  and  views  expressed  in  this 
descriptive  sunnary  lies  with  MR C,  and  not  with  the  author  of 
this  report. 


SOLVING  QUADRATIC  PROGRAMS  BY  AN  EXACT  PENALTY  FUNCTION 

Shih-Ping  Han+ 

1 .  INTRODUCTION 

We  are  concerned  with  the  following  quadratic  program¬ 
ming  problem 


T  T 

min  Jjx  Mx  -  c  x 
s.t.  Ax  <  b  , 


(QP) 


where  M  and  A  are  nxn  and  mxn  matrices  respectively. 
For  convenience,  we  consider  here  only  inequality  constraints. 
The  extension  of  our  results  to  problems  with  equality  con¬ 
straints  is  apparent  and  has  no  major  difficulty.  We  also 
assume  that  the  matrix  M  is  positive  definite.  However, 
for  some  type  Of  semidefinite  matrices,  the  theory  in  this 
paper  can  also  go  through  and  the  proposed  method  is  appli¬ 
cable  . 

Among  the  methods  for  quadratic  programming,  the  class 
of  gradient  projection  methods  have  been  widely  used  and 
studied  [1,2, 3, 6, 7].  The  basic  steps  of  this  type  of  methods 
can  be  described  as  follows.  At  a  feasible  point  x,  we  find 
the  index  set  J  «  {jla^x-b^}  of  active  constraints  at  x 
and  find  the  point  x^  that  solves  the  following  equality 
constrained  quadratic  programming  problem 
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min  %zTMz  -  cTz 
z 


s • t •  AjZ  ®  bj  f 


(1.1) 


where  Aj  and  bj  are  the  submatrix  and  the  subvector  of 
A  and  b  corresponding  to  the  index  set  J  respectively, 
A  search  direction  d  is  then  determined  by 


d  -  X(J>  -  X  . 


The  direction  d  can  also  be  obtained  by  projecting  the 
Newton  direction  -M-1Vf(x)  onto  the  plane  {z|A_z«b_}  . 

U  J 

More  specifically,  we  have 

d  *  -PM“17f(x) 

where  P  *  I  -  m“1Aj(Ajm_1Aj)”1Aj  is  the  projection  matrix 
with  weight  M . 

Once  the  direction  d  is  chosen,  a  new  point  x  ■  x  +  Xd 
is  computed  by  taking  a  stepsize  X  that  is  the  largest  num¬ 
ber  in  [0,1]  keeping  x  +  Xd  feasible.  This  stepsize  proce¬ 
dure  can  be  viewed  as  a  special  case  of  the  following  scheme: 

P(x  +  Xd,a)  *  min  P(x+Xd,a)  , 

X>0 

where  P  is  any  exact  penalty  function  studied  by  Han  and 
Mangasarian  [4]  and  defined  as 

P(x,o)  :»  %xTMx  -  cTx  +  a ||  (Ax-b)  + 1|  .  (1.2) 

Here,  ||*||  is  an  arbitrary  but  fixed  norm  in  K01  and 
(Ax-b)+  is  an  m-vector  with  max  {a^x-b^O}  as  its  i-th 
component.  If  we  prefer  to  maintain  feasibility  of  our 
generated  points  then  we  may • set  the  penalty  parameter  a 


the  value  infinity.  The  resulting  procedure  is  reduced  to 
the  one  used  in  a  usual  gradient  projection  method. 

To  maintain  feasibility#  we  need  a  starting  feasible 
point#  which  is  usually  computed  by  a  linear  programming  tech¬ 
nique.  Because  the  degree  of  complexity  of  linear  programming 
is  about  equivalent  to  that  of  quadratic  programming#  the 
effort  in  finding  a  starting  point  is  unproport ionally  large. 
Moreover,  because  the  objective  function  is  completely  ignored 
in  the  process#  the  obtained  starting  point  is  often  far  from 
our  solution.  Furthermore#  we  sometimes  have  a  good  but  in¬ 
feasible  estimate  of  a  solution#  as  often  in  the  case  that  we 
solve  a  sequence  of  quadratic  programs  to  tackle  a  general 
nonlinear  programming  problem.  It  is  difficult  to  fully  ex¬ 
ploit  this  useful  information  if  we  insist  on  maintaining 
feasibility. 

In  this  paper  we  study  a  gradient  projection  method  that 
allows  the  penalty  parameter  o  to  have  a  finite  value  in 
the  computation  of  stepsizes.  By  doing  so  the  feasibility 
requirement  for  the  generated  points  is  relaxed  and  a  starting 
point  no  longer  needs  to  be  feasible.  This  approach  can  also 
be  viewed'  as  taking  the  objective  function  into  consideration 
in  the  process  of  finding  a  feasible  starting  point  and  com¬ 
bining  the  usual  two  phases  in  a  gradient  projection  method 
into  one. 

lie  also  incorporate  into  this  method  a  procedure  in  se¬ 
lecting  active  constraints  so  that  the  generated  points  may 
not  always  move  along  the  edges  of  the  feasible  region.  If 
the  original  problem  is  feasible  and  the  gradients  of  active 


?3- 


constraints  at  any  feasible  point  are  linearly  independent, 
then  it  is  shown  that  the  method  can  find  a  solution  in  a 
finite  number  of  steps .  Computational  results  show  that  the 
number  of  steps  is  almost  always  less  than  the  number  of  con¬ 
straints. 

We  describe  the  method  in  Section  2  and  study  its  basic 
properties  in  Section  3.  Convergence  theorems  are  given  in 
Section  4.  Section  5  contains  some  computational  results. 


2 .  THE  METHOD 


ro  facilitate  our  description  of  the  method  we  first  in¬ 
troduce  some  notation.  Given  a  point  x,  we  use  J+(x)  to 
denote  the  index  set  {j|ajX>bj}  and  use  Jq(x)  to  denote 
{j|aTx  =  bj>  and  J (x)  =  J+(x)  u  Jg  (x)  .  Notice  that  J(x) 
is  just  the  index  set  of  active  constraints  at  x .  When  the 
reference  of  the  point  x  is  clear  from  context,  we  may  drop 
the  symbol  x  from  J+(x),  Jg(x)  and  J(x)  and  simply  use 
J+,  JQ  and  J,  respectively. 

Given  an  index  set  I  c  {l,2,...,m},  we  are  interested 
in  a  solution  of  the  following  equality  constrained  problem 

min  %xtmx  -  cTx 
x 

s.t.  AjX  »  bj  . 


However,  the  above  problem  may  be  infeasible  for  some  index 
set  I  .  Therefore,  we  prefer  to  consider,  instead,  the 
following  problem 


T  T 

min  Jjx  Mx  -  c  x 
x 

•  T  T 

S.t.  AjAjX  ■  Ajbj  . 


(2.1) 


Here,  the  minimization  is  over  the  set  of  all  least-squares 

solutions  of  the  system  AjX  •  bj  and  the  problem  is  always 

feasible.  In  practice  one  would  not  compute  the  n  x  n  ma- 
T 

trix  AjAj  ,  but  instead  would  make  use  of  a  QR  decomposi¬ 
tion  o!  Aj  .  If  the  matrix  M  is  positive  definite,  as  is 
assumed  in  this  paper,  there  existr  a  unique  solution  to 
(2.1).  We  denoto  this  solution  by  x*1*  .  If  v*1*  is  a 
Lagrange  multiplier  vector  of  x*1*  ,  we  define  an  m-vector 


V 

— 

e  ! 

H  j, 

O'  I 

*<  1 

V 

the  pair 

(x(I> 

,u(I 

(a) 

Hx(I) 

-  c 

(b) 

AjAjX 

(I)  : 

(c) 

“j1’ 

=  0 

We  assume 

that 

the  : 

■  :Sg 


and  u. 


+  A^u'11  -  0 

■  AIbI 


0  for  j  4  I 


0  for  j  4  I  .  Clearly, 


(2.2) 


We  assume  that  the  multiplier  vector  v'  '  is  chosen  in  a 
consistent  way  when  there  are  more  than  one  such  vectors.  By 

so  doing  the  vector  u  ^  is  also  uniquely  determined  by  the 
index  set  I  . 

From  a  result  in  [4],  for  any  vector  norm  ||  *  II  in 
and  for  a  sufficiently  large  number  a,  a  solution  of  quad* 
ratic  program  (QP)  is  also  a  minimum  point  of  the  exact  pen¬ 
alty  function  P  defined  in  (1.2).  In  this  paper  we  use  the 
penalty  function  with  the  2-norm,  because  it  is  compatible 
with  least-squares  solutions  of  the  system  AjX  *  bj  . 

In  the  method  we  improve  our  estimate  x  of  a  solution 
by  finding  a  descent  direction  d  for  the  exact  penalty 
function  P  and  search  along  this  direction  for  a  new  point 
with  a  lower  penalty  function  value.  To  obtain  such  a  direc¬ 
tion  d,  we  consider  the  index  set  J  ■  J(x)  and  compute 
the  pair  (x^,u^)  .  If  (x*J^,u^)  is  not  a  Karush- 
Kuhn-Tucker  pair  of  the  original  problem  (QP)  but  u^J^  £  0 


for  all  i  c  Jq(x),  then  we  set  d 


xtJ>  -  X. 


If  u<J)  <  0 


for  some  i  e  Jq (x) ,  we  drop  one  of  the  constraints  with  a 
negative  multiplier,  say  uj^ ,  and  set  I  ■  J\{j)  .  Com¬ 
pute  (x(I),u(I))  for  this  new  index  set  I  and  set 
d  ■  x*1^  -  x  .  We  repeat  this  deleting  process  until  either 


It  is 


X(I)  ^  X(J)  or  U(I)  >  0  for  all  i  c  JQ(X)  n  I  . 
noted  here  that  if  x  is  feasible  and  the  vectors  {a^}, 
i  €  J(x) ,  are  linearly  independent,  then  no  more  than  one 
constraint  can  be  deleted. 

It  is  observed  that  the  direction  x^  -  x  is  not  al¬ 
ways  better  than  -  x  .  Therefore,  we  set  d  =  x^  -  x 

instead  of  d  =  x^  -  x  when  x  is  feasible,  x^  ^  x 
and 

Js(x(J)-x(I))TM(x(J)-x(I))  <  a||  (AxlI)-b)  ||2  .  •  (2.3) 

This  condition  is  equivalent  to  that 

P(xU),a)  >  Js(x(J))TMx(J)  -  cTx(J)  ,  (2.4) 

and  it  is  proved  later  that  it  helps  the  algorithm  to  ter¬ 
minate  . 

We  sum  up  the  procedure  for  generating  the  direction  d 
at  a  point  x  as  follows: 

1.  Find  J+  =  { j |aTx  >  b^}  , 

J0  *  Olaixssbi>  ' 

J  *  U  J  q  • 

2.  Set  I  -  J  . 

3.  Compute  (x^,  u*1^)  .  If  it  is  a  Karush-Kuhn-Tucker 
pair  for  (QP) ,  then  stop. 

4.  If  x(I)  f  x(J)  then  go  to  7. 

5.  If  uj1*  £  0  for  all  j  «  JQ  n  I  then  go  to  7. 

€.  Find  an  index  j  such  that 
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ujX)  *  min  {u<1}  |u<x)  <  0,  i  e  Jq  n  1} 

and  set  I  =  I \ { j }  .  Go  to  3. 

7.  If  x  is  feasible,  x^  ?  x  and  condition  (2.3)  is 
satisfied,  then  set  d  =  x^  -  x;  otherwise, 
d  =  x(I)  -  x  . 

In  the  sequel,  we  use  I  to  denote  the  index  set  that 
produces  the  direction  d .  Therefore  I  is  equal  to  J 
when  there  is  no  constraint  deleted.  But,  the  index  set 
I  always  satisfies  J+  £  i  c  j  . 

Once  the  direction  d  is  obtained,  we  then  update  the 

—  —  Q 

penalty  parameter  a  by  a  =  10  a,  where  B  is  the  smallest 
nonnegative  integer  that  satisfies 

lO^HAjdllj  >  ||n(I,||2||(AX-b)+||2  .  (2.5) 

It  is  proved  later  that  the  penalty  parameter  remains  bounded 
if  problem  (QP)  is  feasible,  and  that  this  method  of  choosing 
the  penalty  parameter  makes  d  a  downhill  search  direction 
of  the  penalty  function  P  defined  in  (1.2). 

.  A  new  point  x  =  x  +  Xd  is  then  computed  by  doing  a 
line-search  on  the  function  P(«,a);  that  is, 

P(x  +  Xd,a)  =  min  P(x  +  Xd,a)  .. 

X>0 

We  note  here  that  the  function  P(*,a)  is  strict  convex  and 
easy  to  evaluate;  hence,  there  are  no  difficulties  in  the 
line-search  computation. 
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3.  BASIC  PROPERTIES 


In  this  section  we  study  some  basic  properties  of  the 
method.  We  first  establish  the  result  that,  when  problem 
(QP)  is  feasible,  a  point  x  is  its  Karush-Kuhn-Tucker  point 
if  and  only  if  the  direction  d  generated  from  the  method  is 
zero.  We  need  some  lemmas. 


Lemma  3.1:  Let  B  and  E  be  pxn  and  qxn  matrices  and 

1c 

{h  }  and  {g  }  be  sequences  of  p-vectors  and  q-vectors,  re- 

k  k 

spectively.  If  the  system  By  *  h  ,  Ey  <  g  is  consistent 
for  each  k  and  if  h  -*■  h  and  g  g  then  the  system 

By  =  h,  Ey  <,  g  is  also  consistent. 


Proof :  Notice  that  By  =  h,  Ey  £  g  is  consistent  if  and 
only  if 

[2"*c 

where  S  is  the  space  spanned  by  the  columns  of  the  matrix 

and  C  is  the  closed  convex  cone  generated  by  the  col- 

urns  of 

By  our  assumption,  for  each  k  we  have 


c  S  +  C  . 


1c 

Because  S  +  C  is  a  closed  set  and  h 
then  we  have 


h  and 


9/ 


h 

5 


e  S  +  C  , 


which  implies  our  desired  result. 


Lemma  3.2:  Let  Ay  <  b  be  consistent.  Then  there  exists  an 
e  >  0  such  that  for  any  x  £  Rn  and  for  any  index  set  I 
with  J+(x)  c  i  c  J(x),  if  Ajy  *  bj  is  inconsistent  then 
II  Aj  (x  <I}  -x)  ||2  >  E  . 

Proof :  If  the  conclusion  is  not  true,  then  for  each  positive 

v 

integer  k  there  exists  some  x  and  some  index  set  1^  with 
J+(xk)  c  £  J(xk)  such  that 

lim  || A_  (x  k  -xk)|L  *  0 
k-~»  k  * 

k  <V 

but  r  «A_x  -  bT  /  0  .  Since  there  are  only  a  finite 
Jk  Ik 

number  of  index  sets,  without  loss  of  generality,  we  may 
assume  that 

J(xk)  *  J,  J+(xk)  *  J+,  ^  »  I  and  rk  »  r 

for  each  k  .  Hence 


r  *  AjX11*  -  bj  *  AI(x(I)-xk)  +  AjXk  -  bj  . 

Because  AJ(x^I^-xk)  0,  J+(xk)  c  i  and  AjXk  -  bj  >  0, 

we  have  r  yt  0  and  r  £  0  .  Because  x^  is  a  least- 

T 

squares  solution  of  Ajy  »  bj,  we  also  have  A^r  =  0  . 
Consider  the  system 


Ajy  ■  bj  +  r  -  Aj (x(I)-xk) 

V  <  bR  , 


(3.1) 


where  K  *  {l,..*,m}\J.  The  system  (3.1)  is  consistent  be- 
Jc 

cause  x  is  a  solution.  It  follows  from  Lemma  3.1  and 
/  T)  k 

Aj(x'  -x  )  -*•  0  that  the  following  system 
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•  . . . 

V  -  bj  +  r 

V  -  bK 

is  also  consistent  and  has  a  solution,  say  y  .  Let 
Q(y)s«*  %  (Ay-b)^(Ay-b)+ .  Then 

70  (y)  *  AT(Ay-b)  +  =  A^r  *=  0  . 

It  follows  from  the  convexity  of  0  that  y  is  a  minimum 
point  of  0  and  the  minimum  value  of  0  is  0(y)  ■  r  j*  0  . 
This  contradicts  the  assumption  that  the  system  Ay  <,  b  is 
consistent.  Hence  the  proof  is  completed. 

Theorem  3.3:  '  Let  quadratic  program  (QP)  be  feasible.  If  the 
direction  d  generated  from  the  method  is  zero  at  a  point 
x  then  x  is  a  solution  of  problem  (QP) . 

Proof:  If  d  *  x^  -  x  ■  0,  we  want  to  show  that  (x,u^) 
is  a  Karush-Kuhn-Tucker  pair  of  problem  (QP) . 

We  first  show  that  x  is  feasible.  Because 
Aj(x^-x)  *  0  and  J+  c  i  c  j,  it  follows  from  Lemma  3.2 
that  Ajy  =  bj  must  be  consistent.  Thus  we  have 
AjX*1*  =  bj  .  Because  J+  c  i  and  x  -  x*1*  ,  it  follows 
that  AjX  »  bj  ,  J+  *  4*  •  and  hence  that  x  is  feasible. 

It  follows  from  (2.2)  that 


MX  - 


c  ♦  A^u™ 


AjX 


0 


We  also  have  that  A^x  <  bR,  where  K  *  {l,...m}\I,  and 
u<1}  =  0  for  i  e  K  .  Thus,  we  only  need  to  show  that  u^1^  £0. 
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From  the  method,  there  exists  some  i  c  I  with’  u^1^  K  0 

only  if  x(I>  ^  x^J^  .  It  follows  from  AjX*1^  -  AjX  -  bj 

and  I  £  J  that  x^  also  solves  the  problem 
T  T  i 

min  {Jjy  My-c  y|Ajy=b,}  .  By  the  uniqueness  of  solution,  we 

y  u  w 

have  x(I)  *  x^  .  This  completes  the  proof. 

When  x  solves  problem  (QP) ,  it  is  obvious  that  the 
direction  d  generated  at  x  is  zero.  Hence,  we  have  the 
following  corollary. 

Corollary  3.4:  Let  quadratic  program  (QP)  be  feasible.  A 
point  x  is  a  solution  of  problem  (QP)  if  and  only  if  the 
direction  generated  at  x  is  zero. 

Another  straightforward  consequence  of  Theorem  3.3  is 
contained  in  the  following  corollary,  which  is  very  useful  in 
detecting  infeasibility  of  the  problem  (QP) . 

Corollary  3.5;  If  x  is  not  a  Karush-Kuhn-Tucker  point  of 
problem  (QP)  and  the  direction  d  generated  at  x  is  zero 
then  the  quadratic  program  (QP)  is  infeasible. 

We  next  show  that,  if  problem  (QP)  is  feasible,  then 
the  penalty  parameter  remains  bounded. 

Lemma  3.6:  If  problem  (QP)  is  feasible,  then  there  exists  a 
positive  number  p  such  that  for  any  x  «  Rn  and  for  any 
index  set  I  with  J+(x)  £  I  £  J(x)  , 

>  p||(Ax-b)  +  ||2  . 

Proofs  It  follows  from  A^ (A^x ^-bj)  ■  0  that 

||  (Ax— b)  + 1| |  -  ||AI(*'I)-x)|||  ♦  ||A1xI1>-bI||2  . 
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By  Lenuna  3.2  there  exists  c  >  0  such  that 
||Aj (x^^-x)  ||2  <  e  implies  AjX^  =  bj  .  Hence,  if 
|| Aj (x -x)  (1 2  -  e  then  II  Aj  (x*1*  -x)  ||2  -  ||  (Ax-b)  +  ||2  .  If 
IjAjtx^^-x)  ||2  >  e,  let 

n:=  max  { llAjjX^-bjJlj} 

K 

where  the  maximum  is  taken  over  all  index  sets.  Then 


||(Ax-b)  +  ||f  »  ||AI<x(1>-*>|||  +  ||AIx<I)-bI||| 

<  ||AItxtI>-x)|||  +  n2 

<  (1+2^)  ||Ajtx(I,-x)  |||  . 

£ 


Therefore,  the  lemma  follows  if  we  set 

2  2  ** 

ps=  e/(e  +n  ) 

Theorem  3.7:  if  problem  (QP)  is  feasible  then  the  penalty 
parameter  a  in  the  method  remains  bounded. 


Proof :  By  Lemma  3.6,  HAjdJIj  *  0  implies  ||(Ax-b)  +  ||2  *  0 
Therefore,  we  only  need  to  show  that  when  llA^d^  0  the 
quantity 


||  (Ax-b)  + 1| 2 
l|AI(x(I)-x)  ||2 


remains  bounded  for  any  x  e  Rn  and  for  any  index  set  I 
with  J+(x)  c  i  c  j(x)  . 

The  value  ||u^||2  is  bounded  because  there  are  only 
a  finite  number  of  index  sets  and,  as  mentioned  before,  for 
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each  index  set  I  the  vector  is  uniquely  determined. 

By  the  previous  lemma,  we  have  that  for  some  constant  p 

H  <**"*>> +  11 2  i 

llAjtx'11-*)  ||  2  "  0  ' 

Hence,  the  proof  is  completed. 


Corollary  3.8:  If  the  penalty  parameter  •  in  the 

method  then  problem  (QP)  is  infeasible. 

He  observe  that  the  penalty  function  p(x,a)  is  convex 
and  for  any  direction  d,  its  directional  derivative 
p’(x,a;d)  at  x  exists  and  is  given  by 


p  r(x,a;d) 


T  T 

d  MX  -  c  d  + 


odTAT(Ax-b)+ 
H  (Ax-b)  +  ||  2 


if  x  is  infeasible;  (3.2) 


dTMx  -  cTd  +  a|l  (Ajd)  +  U2 

if  x  is  feasible. 


The  following  two  theorems  show  that  the  direction  d 
generated  from  the  method  is  a  descent  direction  for  the 
exact  penalty  function. 


Theorem  3.9:  Let  quadratic  program  (QP)  be  feasible  and  let 
x  c  Rn  be  an  infeasible  point  to  (QP) .  If 


>  ||U(1)||2  , 

2  HVH* 


then 


p  *(x,a;d)  <  -dAMd  . 


Proof;  Using  (2. 2. a)  and  (3.2)  to  eliminate  c,  we  got 


-»*  «(► 


* 


.T.T, 


p'(x,md)  -  -d»Hd  -jVuj11  ♦  .  — (AX--^  . 

11  llUx-Mj, 


(3.3) 


On  the  other  hand,  it  follows  from  j+(x)  c  i  c_  j(x)  and 


-  0  that 


dTAT  (Ax-b) 


dTA^  (AjX-^) 


dTA^(AJx(I,-bI-AId) 


-  -d^Ajd 


-  -IIVH2 


Thar* for a,  if 


#T.  ||  (Ax-b).  || , 

i  ii»<I,n2i;- — , 
2  IIVIIj 


than 


odTAT(Ax-b) 


1  <  iivMu<i>H2  “ 


||  (Ax-b)  + 1|  2 


llAjdH 


||  (Ax-b)  + 1|  2 


<  0  . 


This,  in  conjunction  with  (3.3),  implies  our  result. 


Theorem  3.10:  If  x  is  a  feasible  point  of  problem  (QP)  and 
the  vectors  (a^>,  i  <  J(x),  are  linearly  independent,  then 
for  any  a  , 

p’(x,a;d)  <  -dTMd  . 

Proof:  Using  (2. 2. a)  and  (3.2)  to  eliminate  c,  we  have  that 
p  *(x,a;d)  -  -dTMd  -  dTAjUjJ*  ♦  a||(Ajd)f||2  . 


I 


T 

We  only  need  to  show  that  Ajd  ■  0  and  a^d  <  0  for  any 
j  c  J\I,  where  J  ■  J(x)  . 

Because  x  is  feasible,  we  have  A^x  »  bj  .  Hence, 
it  follows  from  I  £  J  that  the  system  Axy  *  bj  is  con¬ 
sistent  and  AgX^  ■  bj  .  Therefore,  we  have  that 

Ajd  -  Ajtx^-x)  -  0  . 

T 

To  show  that  a^d  <,  0  for  each  j  c  J\I,  we  first  show 

that  the  set  J\1  can  not  contain  more  than  one  indices  if 

{a^},  i  «  J,  are  linearly  independent.  Assume  that  there 

are  more  than  one  indices  in  J\I  .  Let  q  and  p  be  the 

first  and  the  second  indices  deleted  from  J,  respectively. 

Then,  by  the  method,  the  point  x ^  is  a  Karush- Kuhn- Tucker 

point  of  the  problem  min  {%yTMy-cTy|A-.y«b  ,  aTy-b  ,  aTy<*b  } 

a  K  p  p  q  q 

and  also  of  the  problem  min  {>iyTMy~cTy  |  Aj^y-b^  a^y-bp}  , 
where  K  -  J\{p,q}  .  Hence,  there  exists  vectors  u,  v  and 
numbers  a,  0  and  y  such  that 

-  c  +  aHu  +  aa„  +  Ba^  »  0 
*  P  q 

and 

Mx*J*  -  c  +  A^v  +  yOp  -  0  . 

Here,  we  have  6  <  0  because,  if  not,  the  index  q  can  not 
be  deleted  from  J  .  Then,  eliminating  the  term  Mx^J*  -  c 
from  the  above  two  equations,  we  get 

aJJ(u-v)  +  (a-y)ap  +  6ag  -  0  , 

which  contradicts  the  assumption  that  {a^},  i  «  J,  are 
linearly  independent. 


If  I  m  j  then  there  is  nothing  to  prove.  Let  j 
the  only  index  in  J\I.  Then  it  follows  from  Theorem  1 
[6]  that  .  <  0  .  Hence  we  have  that 


* 


.*(x(1>-x(J>)  ♦  .*(X(J>-X> 
,*<x(I>-x<J)>  <0. 


be 

.9  in 


This  completes  our  proof. 


4.  FINITE  CONVERGENCE 

In  this  ssction  we  show  that,  under  suitable  conditions, 
the  method  generates  from  any  starting  point  a  solution  of 
Problem  (QP)  in  a  finite  number  of  iterations. 

The  proof  of  finite  convergence  for  the  method  is  fairly 
long.  To  make  it  easier  to  understand,  we  outline  our  ap¬ 
proach  below.  The  proof  is  done  by  contradiction.  If  the 
method  generates  an  infinite  sequence  {x  } ,  then  we  show 
that  the  sequence  enters  from  infeasible  region  into  feasible 
region  for  only  a  finite  times.  This  implies  that,  after 
finite  iterations,  either  all  the  points  will  remain  infea¬ 
sible  or  all  become  feasible.  Then  we  get  a  contradiction  by 
showing  that  either  case  can  not  occur. 

For  convenience,  we  give  the  following  definition. 

Definition  4.1:  Problem  (QP)  is  said  to  be  regular  if  it  is 
feasible  and  for  any  feasible  point  x,  the  vectors  {a^}, 
i  c  J(x) ,  are  linearly  independent. 

We  first  present  a  lemma,  which  is  a  straightforward 
consequence  of  Theorem  3.7. 

Lemma  4.2;  If  quadratic  program  (QP)  is  feasible  then  there 
exists  a  number  a*  and  an  integer  £  such  that  the  penalty 
parameter  ■  a*  for  all  k  >,  £  . 

Theorem  4.3:  If  quadratic  program  (QP)  is  regular,  then 
is  greater  than  or  equal  to  one  for  only  a  finite  number  of 
values  of  k  . 

Proof i  By  Lemma  4.1,  after  finite  iterations  and  for  a  fixed 
a*,  the  value  of  function  P(*,a*)  maintains  strictly 
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monotone  decreasing.  If  d  ■  -  x  ,  x  ■  x  +  Xd  and 

X  >  1  then  at  least  one  of  the  two  inequalities 

P(x,o*)  >  P(x^  ,<x*) 

and 

P(x{1)  ,a*)  >  P(x,a*) 

is  satisfied.  Because  there  are  only  finite  index  sets  I, 
the  above  inequalities  can  happen  for  only  a  finite  number  of 
times.  Hence,  the  desired  result  follows. 

Lemma  4.4:  If  x  is  infeasible  and  J+ (x)  =  I  c  J(x)  then 
for  any  X  in  [0,1),  the  point  (l-X)x  +  Xx^1*  is  infea¬ 
sible. 

Proof:  If  it  is  not  true  then  there  exists  a  X  in  [0,1) 
such  that  the  point  x  *  (l-X)x  +  Xx^  is  feasible;  that 
is,  a^x  <  b^  for  each  i  .  Clearly,  X  can  not  be  zero 
and  x  =  x  +  Xd  . 

For  any  i  we  have  that 

m  1  T  a  1  T 

aid  *  Xai  ^x”x*  —  X  tbi“aix)  • 

T 

Therefore,  it  follows  that  a^d  <0  if  i  e  J+(x)  and 
T 

ajd  <  0  if  i  c  I\J+(x)  .  On  the  other  hand,  we  have  that 
a*x(I)  -  bi  *  a*(x+(l-X)d)  -  bL 
<  (l-X)a^d  . 

This,  in  conjunction  with  the  fact  that  J+(x)  ?  ♦,  implies 


that 


T  (I) 

But,  this  inequality  contradicts  that  AjCAjX'  '-bj)  *  0  • 
This  completes  our  proof. 

V 

It  follows  from  Lemma  4.4  that  if  x  is  infeasible  and 

k+1 

x  is  feasible  then  the  stepsize  must  be  no  less  than 

one.  This  result,  in  conjunction  with  Theorem  4.3,  implies 
the  following  lemma. 

Lemma  4.5:  If  quadratic  program  (QP)  is  regular  and  {x  } 

is  a  sequence  of  points  generated  by  the  method,  there  are 

k  k+1 

only  a  finite  number  of  consecutive  pairs  (x  ,x  )  in  the 

k  k+1 

sequence  with  x  infeasible  and  x  feasible. 

Theorem  4.6:  Let  quadratic  program  (QP)  be  regular  and  {x  } 
be  a  sequence  of  points  generated  by  the  method,  then  one  and 
only  one  of  the  following  statements  holds: 

)c 

(a)  The  sequence  {x  }  terminates  at  a  solution  of  (QP)  in 
a  finite  number  of  steps; 

a  V 

(b)  There  exists  k  such  that  x  is  infeasible  for  all 

k  >  kj 

«  V 

(c)  There  exists  k  such  that  x  is  feasible  for  all 

k  >  k. 

In  the  rest  of  this  section  we  establish  a  finite  con¬ 
vergence  theorem  for  our  method  by  showing  that  statements 
(b)  and  (c)  in  Theorem  4.6  actually  can  not  hold. 

We  first  deduce  a  contradiction  when  Statement  (b)  in 
Theorem  4. 6. is  true.  Our  deduction  depends  on  that  the 


condition  AjX  ^  *  bj  always  holds  af.ter  a  certain  itera¬ 
tion.  This,  in  turn,  is  derived  from  that  the  condition 
holds  when  the  corresponding  point  x  is  in  a  slightly  en¬ 
larged  feasible  region 

ty|Ay<b  +  ne}  (4.1) 

where  e  is  the  vector  of  ones  and  n  is  a  suitable  posi¬ 
tive  number.  To  determine  the  number  n  we  first  define 

5^:=max  {a^x^-b^|i  e  1} 

6  :*  min  {6(I) |l  « ft} 


where 


ft:=  {l|AIxtI)  b ^  and  the  system 
¥^bi  is  consistent}  . 

Here,  we  use  the  convention  that  6  =  «  when  ft  =  <{>  . 
the  number  n  in  (4,1)  is  determined  by 


Then 


6 

/2m 


(4.2) 


We  need  to  show  that  rj  is  positive,  which  follows  from  the 
following  lemma. 

Lemma  4.7:  If  left  then  6 ^  >  0  . 

Proof:  Let  r:»  AjX^-bj  .  If  I  e  ft  and  <  0  ,  then 

r  i*  0  and  r  <_  0  .  Let  x  be  any  point  satisfying  the  sys¬ 
tem  Ajy  £  bj  .  Then 

A^x-x*1*)  >  -r  . 
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I 


Premultiplying  the  above  inequality  by  the  nonzero  and  non¬ 
negative  vector  - r ,  we  get 

>  rTr  >  0  . 

T 

This  contradicts  that  A.j.r  =  0  ,  and  hence  the  lemma  is 
proved. 

Corollary  4.8:  The  number  n  defined  in  (4.2)  is  positive. 


Theorem  4.9:  If  x  e  X_  and  J. (x)  c  i  c  j(x)  then 
-  n  +  —  — 

AjX11’  =  . 


Proof:  If  it  is  not  true  then  left  and 


(AIx(I,-bI)T(AIx(I)-bI)  >  52  . 


It  follows  from  x  e  X  that 

n 

(AjX-^) T  (AjX-b)  <  mn2 


This  contradicts  that  x^  is  a  least-squares  solution  of 
V  m  bI  ‘  Hence  the  theorem  is  proved. 

We  next  show  that  there  are  at  most  a  finite  number  of 
iterations  in  which  the  condition  A,x^  =  b,  does  not  hold. 


Lemma  4.10:  Let  x  be  any  point  in  R  .  If 

J+(x)  c  i  £  j(x)  and  AjX^  ^  bj  then  for  each  X  c  [0,1], 

(l-A)x  +  Xx(I)  i  . 

Proof :  Suppose  that  it  is  not  true  and  for  some  X  in 
[0,1]  the  point  x  =  (l-X)x  +  Ax^  is  in  X^  .*  By  Theorem 
4.9  and  the  definition  of  n#  the  number  X  must  be  in  the  open 
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interval 


(0,1).  Let  K:=  {ijaTx^b^,  i  c  1}  and  let  j  be 


an  index  in  I  that  satisfies 

=  a*x(I>  -  bj  . 

T  IT) 

It  follows  from  Lemma  4.7  that  atx  ’  -  bj  >  0  ,  which  im¬ 
plies  that  j  e  K  .  Therefore,  because  x  e  X^  ,  we  have 
that 


l  (aTx-b . ) 2  <  miV 
i«K  11  , 

_  6^ 


<  (6m)2 

<  l  (a^x(1)-b  )2  . 
ieK  1  1 


On  the  other  hand,  for  i  £  I\K  we  have  that 

T  (I)  .  .  _Ts  y.  s 

Sl^X  "  ^  cl^X  "  ^  U  * 


Hence,  it  follows  that 


I  (aTx-bi) 2  <,  l  (a?x 
i€l\K  1  1  icI\K  1 


Therefore,  we  have  that  IjAjX-b^l^  <  [(AjX  1  -b.J|2  •  This 
contradicts  that  x^  is  a  least-squares  solution  of 
Ajy  =  bj  and  hence  the  proof  is  completed. 

Theorem  4.11:  If  problem  (QP)  is  regular  then  there  are  at 
most  a  finite  number  of  iterations  in  which  AjX^  ^  b^.  . 

Proof:  Suppose  that  at  a  certain  iteration  Aj.x^1^  ?  bj  , 
then  by  Lemma  4.10  we  have  that  the  whole  line-segment 
(x,x^]  is  contained  in  the  complement  set  of  X^  .  Define 

Y:*  (xjj)  n  { y  1 P  (y,ci*)  <  P(xk,a*)} 


where  (X^)  is  the  closure  of  X^  and  k  is  a  positive 
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integer  such  that  for  each  k  >,  k,  ®  a*  and  X^  <  1  . 

The  compact  set  Y  is  contained  in  the  infeasible  region,  in 
which  P  is  continuously  differentiable,  hence  P  has  uni¬ 
formly  continuous  derivatives  on  Y .  Thus  it  follows  from 
the  proof  of  Theorem  14.2.7  in  [5]  that  there  exists  a  forcing 
function  a  such  that 

P (x,a*)  -  P (x,a*)  >  0(lVp(x,  » ) Td | }  #  (4.5) 

l|d||2 

The  above  inequality  holds  whenever  A^x^  ^  b^  .  The  se- 
quence  {P(x  ,a*)}  is  monotone  decreasing  and  bounded  below 
and  hence  convergent.  If  A_x^  ?  bj  for  an  infinite  number 
of  times  then  it  follows  from  (4.5)  and  -VP(x,ct*)T  >  dTMd 
that  for  any  e  >  0  there  exists  k  £  k  such  that 

Uk) 

A-  x  *  ¥  b_ 

Ak 

l|dk||2  <  *  . 

But  this  is  impossible  because  it  contradicts  the  conclusion 
of  Lemma  3.2.  Hence  our  proof  is  completed. 

We  are  now  ready  to  show  that  Statement  (b)  of  Theorem 
4.6  can  not  hold. 

1c 

Theorem  4.12:  If  problem  (QP)  is  regular  and  {x  }  is  an 

infinite  sequence  generated  by  the  method  then  there  exists 
-  k 

no  k  such  that  x  is  infeasible  for  all  k  >  k  . 

Proof ;  Suppose  the  conclusion  is  not  true  and  there  exists  a 

w  V 

positive  integer  k  such  that  x  is  infeasible  for  each 
k  >  k  .  By  Theorem  4.3  and  4.11  we  can  choose  k  such  that  in 
any  iteration  with  k  >  k,  we  have  X  <  1  and  AjX*1*  «  bj  . 


We  want  to  show  that  in  any  iteration  with  k  >  Jc  the  set 
J+(x)  is  a  proper  subset  of  J+(x)  .  This  certainly  implies 
a  contradiction  because  there  are  only  a  finite  number  of  in¬ 
dex  sets. 

Let  i  e  J+(x),  then  it  follows  from  a^x  >  b^  and 
a?x^  =  b^  that 

a?x  -  (1-X) a?x  +  XaTx(I) 

^  b^  • 

Therefore,  we  have  J+(x)  £  J+(x)  .  To  show  J+(x)  j  J+(x) 
we  consider  function  defined  by 

?1(x,a*)i=  3sxTMx  -  cTx  +  a*  ||  (AIx-bI)  +  ||2  . 

It  is  convex  and  the  function 

<fr(X):=  Pj (x  +  Xd,a*) 

decreases  monotonically  for  0  <  X  ^  1  .  In  order  to  prove 
the  monotonicity  one  obtains  from  (3.2)  and  (2. 2. a)  the  bound 

$'(1-)  =  dTMx(I)  -  cTd  -  a*  ||  ("AjdJ^I^ 

-  -dTA*u(I)  -  a* 1 1 Ajd || 2 

<  (|lu<1)  ||~a*)  II Ajd || 2  . 

Because  of  the  lower  bound  on  a  that  is  given  in  the  state¬ 
ment  of  Theorem  3.9,  and  because  the  proof  of  Lemma  3.6  shows 
that  ||  (Ax-b)  + 1| 2  is  no  less  than  HAjd^  ,  it  follows  that 
♦’(l-)  <  0  ,  which  establishes  the  monotonicity.  Therefore, 
we  deduce  from  the  continuity  of  derivatives  in  the  infeasible 
region  that  the  value  P(x,a*)  is  not  the  same  as  the  value 

of  Pjtx.o*)  .  It  follows  that  J-K(x)^J+(x)  and  hence  the 
proof  is  completed. 

We  now  show  that  Statement  (c)  in  Theorem  4.6  also  can 
not  hold. 
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Theorem  4.13:  If  problem  (QP)  is  regular  and  {x  }*  is  an 
infinite  sequence  that  is  generated  from  the  method,  then 

w  If  mm 

there  exists  no  k  such  that  x  is  feasible  for  all  k  £  k . 
Proofs  Suppose  that  there  exists  a  k  such  that  for  each 

•  lr 

k  £  k,  the  point  x  is  feasible.  By  Lemma  4.2  and  Theorem 
4.3,  we  may  assume  that  =  a*  and  <  1  for  all  k  >  k  . 

We  will  establish  a  contradiction  by  showing  that  for 
k  £  k,  each  set  in  the  sequence  {J(x  )}  is  different.  Let 
Card(J)  denote  the  cardinal  number  of  the  index  set  J  .  We 
first  show  that  for  k  £  k,  the  sequence  of  numbers 
{Card(J(x  ))}  is  nondecreasing. 

Let  x  be  a  feasible  point  and  d  be  the  direction 
generated  at  x  and  d  =  x^  -  x  for  some  index  set  I  . 

As  in  the  proof  of  Theorem  3.10,  it  follows  from  the  regular¬ 
ity  of  the  problem  (QP)  there  is  at  most  one  constraint  de¬ 
leted  and  we  have  that  I  =  J  or  I  =  J\{j}  for  some  j  e  J  . 
It  follows  that  AjX  »  bj  and  I  £  J  that  the  system 
V  -  bj  is  consistent  and  AjX^  *  bj  .  Therefore,  if 
x  *  x  +  Xd  and  I  <  1  then  at  least  a  new  constraint  is 
added  to  I  and  we  have  that 

Card(J(x))  >,  Card(J(x)}  +1  if  I  *  J 

# 

Card(J(x) )  >  Card(J(x) )  if  I  »  J\{ j >  . 

This  implies  that  for  k  >  R,  the  sequence  of  numbers 

{Card(J(x  ))}  is  nondecreasing. 

We  now  show  that  for  k  £  E,  the  sets  {Jlx^))  are  all 

different.  It  is  obvious  that  the  index  set  Jtx)  can  not 

k 

be  repeated  in  the  sequence  {J(x  )}  if  I  «  J  .  This  is 
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because  that  Card(J(x))  >  Card(J(x))  +  1  and  the  sequence 
{Card(j(x  ))>  is  nondecreasing.  We  only  need  to  consider 
the  case  I  *  J\{j>  . 


If  I  »  J\{j},  then  by  (2.4)  we  have  that 
%(xtJ)>TMx<J>  -  cTxtJ>  >  P(*(I>,a*>  . 


By  the  line-search,  we  also  have  that 


(!) 


P(x  ,a*)  >  P (x,a*) 


Let  x  be  the  point  that  solves 


min  { JsyTMy-cTy  |  Ajy  *  bj ,  J^y  <  bR} 


where  K  *  {l,...,m)\J.  Then, 


P(x,o*)  =  *jxTMx  -  cTx  >  %(x(J))TMx(J)  -  cTx(J) 


>  P(xtIJ  ,o*) 


,>  P(x,a*)  . 


If  z  is  any  point  that  comes  after  the  point  x  in  the  se- 


K 

quence  {x  }  with  J(z)  *  J(x),  then  it  follows  from  the 


feasibility  of  z  and  AjZ  ®  bj  that 


P(z,o*)  *  JjzTM z  -  cTz  >_  *jxTMx  -  cTx 


(4.6) 


>  P(x,o*)  . 


Because  z  /  x  and  z  comes  after  x  in  the  sequence 
k 


It  » 

(x  ),  we  have  that  P(x,o*)  >  P(z,o*),  which  contradicts 
(4.6).  The  proof  is  then  completed. 
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We  now  conclude  this  section  with  the  following  finite 
convergence  theorem. 

Theorem  4.14:  If  quadratic  program  (QP)  is  regular  then  the 
method  produces  the  solution  of  (QP)  from  any  starting  point 
in  a  finite  number  of  steps. 

Proof :  This  immediately  follows  from  Theorems  4.6,  4.13  and 


5.  COMPUTATIONAL  RESULTS 


Some  preliminary  computational  tests  have  been  carried 

out  for  the  method  in  the  Cyber  175  System  at  University  of 

Illinois-Urbana.  The  test  problems  are  randomly  generated 

but  positive  definiteness  of  the  matrix  M  is  maintained. 

The  starting  point  is  chosen  to  be  an  unconstrained  minimum 

point  of  the  objective  function.  A  linear  system  subroutine 

in  LINPACK  is  used  to  solve  the  equality  constrained  problem 

T  T 

(2.1)/  where  the  constraint  A^A^x  *  AJbT  is  taken  care  by  a 

QR  decomposition.  A  point  is  accepted  as  a  solution  when  the 

Karush-Kuhn-Tucker  conditions  are  satisfied  within  the  tol- 
-12 

erance  10  .A  constraint  is  considered  to  be  active  if  it 
is  satisfied  within  10~7.  We  note  here  that  the  Cyber  175 
has  a  48  bit  mantissa. 

In  the  following  tables,  n  is  the  dimension  of  vari¬ 
able  vector  and  m  is  tne  number  of  inequality  constraints. 
The  numbers  are  the  numbers  of  calls  for  a  linear  system  sub¬ 
routine,  rather  than  those  of  iterations,  which  are  usually 
smaller. 


TABLE  1 


X 

5 

10 

15 

20 

25 

30 

35 

40 

45 

50 

50 

3 

14 

16 

14 

14 

23 

18 

19 

23 

33 

45 

5 

5 

15 

9 

23 

16 

22 

25 

27 

36 

40 

8 

10 

12 

16 

30 

17 

30 

34 

20 

34 

35 

5 

6 

14 

9 

28 

18 

26 

15 

31 

19 

30 

5 

9 

8 

17 

28 

21 

18 

21 

28* 

17 

•Indicates  that  the  problem  is  infeasible. 


TABLE  2 


XI 

5 

10 

15 

20 

t—  -  ■■■ 

25 

30 

25 

D 

6 

3 

ii 

32 

25 

20 

3 

8 

9 

10 

15 

9 

15 

5 

n 

12 

12 

28 

10 

5 

B 

10* 

19* 

10 

18* 

♦Indicates  that  the  problem  is  infeasible. 

'(‘Indicates  that  the  tolerance  10  ^  has  not  been  satisfied 
in  40  iterations. 

According  to  these  results,  it  seems  that  the  number  of 
iterations  are  usually  less  them  the  number  of  constraints 
and  the  efficiency  of  the  method  depends  more  on  m  than  n  . 
Therefore,  the  method  is  recommended  for  problems  with  fewer 
constraints . 

The  method  becomes  very  efficient  when  we  have  a  good 
but  infeasible  estimate  of  solution.  This  makes  it  useful  in 
solving  general  nonlinear  programming  problems. 
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